Fault recovery method in a system having a plurality of storage systems

ABSTRACT

System availability is improved in a second storage system, connected to a first storage system, and having means for virtualizing devices within the first storage system as its own devices. When the virtual storage system or a storage management server detects a fault in the virtual storage system, the management server investigates the range affected by the fault, identifies a device for which measures must be taken, determines a transfer target device which accommodates the performance, reliability, and other attributes of the affected device, and issues a device transfer instruction for the virtual storage system. In the virtual storage system, the data of the device specified by the instruction within the virtual storage system is transferred to a device, specified by the management server, within the system itself, or to a device within another virtual storage system.

CROSS-REFERENCES TO RELATED APPLICATIONS

This application relates to and claims priority from Japanese PatentApplication No. 2004-142179, filed on May 12 2004, the entire disclosureof which is incorporated herein by reference.

BACKGROUND

This invention relates to a storage system comprising storage whichstores data used by computers in a computer system. In particular, thisinvention relates to control technology in a storage system comprisingstorage having means for connecting one or more storage units, and forrendering virtual, as its own device, a device within the connectedstorage unit.

In recent years there has been explosive growth in the volume of datahandled by computers, and as a consequence the capacity of storage unitfor storing data is steadily being increased. As a result, storagemanagement costs account for an increasing fraction of system managementcosts, and the need to lower management costs has become an urgent issuefor system operation.

In order to expand storage capacity, new storage may be introduced intoan existing computer system comprising a computer (hereafter called a“host”) and storage unit. Two such modes of introduction areconceivable, one in which new large-capacity storage unit is introducedto replace older storage unit, and the other in which the new storageunit is used in conjunction with the older storage unit.

In the case of a mode of introduction in which new storage unit replacesold equipment, all the data within the old storage unit must betransferred to the new storage unit. However, ordinarily the data mustbe transferred while continuing data input from and output to a host.

Technology to transfer the data of old storage unit to new storage unit,while continuing data input/output with a host, has for example beendisclosed in JP-A-10-508967.

Here, the data of a first device of the old storage unit is transferredto a second device allocated to the new storage unit, and the accesstarget from the host is changed from the existing first device to thenew second device, so that input/output requests issued from the host tothe existing first device are accepted by the new storage unit.

Read requests issued during the transfer are handled by reading from thesecond device for portions transfer of which has been completed, and byreading from the existing first device for portions transfer of whichhas not been completed. In the case of write requests, duplicate writingto both the first device and the second device is performed.

In a mode of introduction in which old storage unit and new storage unitare used in conjunction, a mode is possible in which both the new andold storage units are connected directly to the host; but control on thehost side is complex.

On the other hand, in for example Japanese Patent Laid-open No.1-283272, a method is disclosed by which a host accesses a disk of afirst storage unit through a second storage unit.

A configuration is employed in which the first storage unit is connectedto the second storage unit, disk addresses of the second storage unitare allocated to disks of the first storage unit, and the host alsoaccesses the disks of the first storage unit through the disk controldevice of the second storage unit.

Upon receiving an input/output request from the host, the second storageunit judges whether the disk being accessed is a disk of the firststorage unit or is a disk within the second storage unit, anddistributes the input/output request to the access target according tothe judgment result.

SUMMARY

By applying the technology disclosed in Japanese Patent Laid-open No.10-283272, that is, technology whereby a storage unit has the hostrecognize a disk of another storage unit connected to itself as its owndisk, a storage system can be constructed in which a plurality ofstorage units, with different attributes such as performance,reliability and cost, can be integrated.

For example, when new storage unit is installed in a computer system, ifthe newly installed new-type storage unit, having the functionsdisclosed in the above-described Japanese Patent Laid-open No.10-283272, is directly connected to the host in a configuration in whichthe old-type storage unit already possessed by the user is connected tothe new-type storage unit, the user can effectively utilize existingresources, and the cost of installation in the system can be reduced.

When constructing a computer system, if the storage system adopts aconfiguration in which a plurality of low-cost, low-functionalitystorage units are connected to high-cost, high-functionality storageunit having functions disclosed in the above-described Japanese PatentLaid-open No. 10-283272, then a hierarchical storage system can berealized in which data is optimally arranged according to the freshnessand value of the data. In such a storage system, a large volume of datasuch as the transaction information and mail logs which occur in thecourse of daily operations, and which although not accessed frequentlymust be preserved for long periods of time for monitoring or otherpurposes, can be stored in the low-cost, low-functionality storage unit,so that storage resources can be utilized effectively.

However, in the above-described storage system, old-type storage unitwhich is the existing resources of the user coexists with low-coststorage unit the purpose of which is to store large amounts of data atlow cost. There is a strong possibility that such storage unit, withcomparatively low reliability, may detract from the reliability of thestorage system and of the entire computer system.

Further, when a storage system is configured by connecting a pluralityof storage units, such connections may be through a network. In thiscase, network faults may result in blockage of access paths.

As stated above, a storage system comprising second storage unit, havingmeans for connecting first storage unit and for rendering virtual adevice within the first storage unit as a device within the secondstorage unit, is often configured integrating a plurality of storageunits with different performance, reliability, cost, and otherattributes. Hence due to the existence of comparatively low-reliabilitystorage unit and to the existence of a network connecting storage unitin such a storage system, there is the problem that the availability ofthe storage system and of a computer system comprising the storagesystem cannot be improved.

In light of the above, availability can be improved in a computer systemhaving second storage unit which has means for connecting first storageunit, and for rendering virtual a device within the first storage unitas its own internal device.

In order to attain this object, a computer system comprises a managementserver, which manages both a first storage unit, and also a secondstorage unit which provides to the host computer as logical devices botha logical device provided by the first storage unit (hereafter called an“external device”) and its own physical device. This management servercomprises transfer source decision means which, based on informationreceived from the second storage unit prognosticating a fault in theabove external device, identifies the range influenced by the fault asthe transfer source, and data transfer instruction means which, based onthe data capacity of the transfer source and on evaluations ofperformance and reliability levels established in advance, determinesthe transfer targets in the storage range of the first storage unitmanaged by itself and the second storage unit, and issues to the secondstorage unit an instruction to transfer the data of the above transfersource to the above transfer target.

Availability can be improved in such a computer system comprising asecond storage unit having means for connecting a first storage unit andfor rendering virtual a device within the first storage unit as its owninternal device.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows one example of the hardware configuration of a computersystem to which a first aspect is applied;

FIG. 2A shows one example of control information stored in storagecontrol memory and in memory, and a program for storage controlprocessing, in the first aspect;

FIG. 2B shows one example of control information stored in the memory ofthe management server of the first aspect, and one example of a programfor storage control processing;

FIG. 3 shows one example of logical device management information in thefirst aspect;

FIG. 4 shows one example of LU path management information in the firstaspect;

FIG. 5 shows one example of physical device management information inthe first aspect;

FIG. 6 shows one example of external device management information inthe first aspect;

FIG. 7 shows one example of storage management information in the firstaspect;

FIG. 8 shows the flow of processing by an input/output requestprocessing program in the first aspect;

FIG. 9 shows the flow of processing by an external device monitoringprocessing program in the first aspect;

FIG. 10 shows the flow of processing by a storage monitoring processingprogram in the first aspect;

FIG. 11 shows the flow of processing by an external device transferinstruction processing program in the first aspect;

FIG. 12 shows the flow of processing by an external device transferprocessing program in the first aspect;

FIG. 13 shows the flow of processing by an external device recoveryprocessing program in a second aspect; and, FIG. 14 shows one example oflogical device management information in the second aspect.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

As aspects of the invention, first and second aspects are explained.

The first aspect is summarized below.

The system assumed in the first aspect is a storage system in which oneor more first storage units are connected, as external storage, to asecond storage unit having external storage connection functions.

Here, the external storage connection functions of the above secondstorage unit are functions by which, upon receiving an access requestfrom the host, the second storage unit judges whether the device forinput/output of the access request is a device existing in a firststorage unit or is a device in the second storage unit itself, and if adevice in a first storage unit, transmits the access request to thefirst storage unit, but if a device in itself, accesses the device.

The first aspect endeavors to provide data integrity, at the time that aprognostication of occurrence of a fault in a device within a firststorage unit is discovered, by transferring data stored in the devicefor which the fault prognostication has occurred to another device.

A server provided to manage storage (hereafter called a “managementserver”) detects the occurrence of prognostications of faults in a firststorage unit, based on anomaly reports from first storage units, and onwarnings from the second storage unit of anomalies in first storageunits. Warnings are issued from the second storage unit based onprognostications of faults in first storage units, detected bymonitoring responses during accessing of first storage units andsimilar.

The management server, after detecting a fault prognostication,identifies devices within the first storage unit of the faultprognostication which would be affected were the fault to occur, anddecides on the device for data transfer, while also selecting the deviceto be the transfer target based on the attributes of the device for datatransfer. Then the data transfer is instructed to the first storageunit.

The second aspect is summarized below.

Similarly to the first aspect, a storage system of the second aspect isconfigured with one or more first storage units connected to a secondstorage unit having external storage connection functions. In the secondaspect, a first storage unit uses a plurality of devices in a RAID(Redundant Array of Independent Disks) configuration, which is providedto the host as a disk device of the second storage unit.

In the second aspect, in addition to a function to endeavor to providedata integrity prior to occurrence of a fault similarly to the firstaspect, the data stored in a device in which a fault actually occurs isrecovered and is transferred to another device.

Similarly to the first aspect, upon receiving a first storage unitanomaly report the management server identifies the range affected bythe anomaly, decides on the transfer source and transfer target, andissues a data transfer instruction to the second storage unit. Further,upon receiving a report of the actual occurrence of a fault, themanagement server utilizes RAID properties to recover the data stored onthe device in which the fault occurred, and issues an instruction to thesecond storage unit to store the recovered data on a device selected asthe transfer target.

First Aspect

The first aspect is explained referring to FIG. 1 through FIG. 12.

FIG. 1 shows one example of the hardware configuration of a computersystem to which the first aspect of this invention is applied.

The computer system comprises one or more host computers (hereaftercalled “hosts”) 100; a management server 110; a fibre channel switch120; storage unit 130; a management terminal 140; and external storageunit 150 a and 150 b (collectively called “external storage 150”).

The hosts 100, storage unit 130 and external storage unit 150 areconnected to ports 121 of the fibre channel switch 120 via the ports107, 131, 151 respectively. The host 100, storage unit 130, externalstorage unit 150, and fibre channel switch 120 are connected to themanagement server 110 via the interface control portions (I/F) 106, 138,157, 123 respectively through the IP network 175, and are integrated andmanaged by storage management software, not shown, which runs on themanagement server 110.

In this aspect, the storage unit 130 is connected to the managementserver 110 via the management terminal 140; however, a configuration maybe employed in which the storage unit 130 is connected directly to theIP network 175.

The hosts 100 are computers which execute applications and access thestorage unit 130, and each comprise a CPU 101, memory 102, storagedevice 103, input device 104, output device 105, interface controlportion 106, and port 107.

The CPU 101 reads the operating system, application programs, and othersoftware stored on a hard disk, magneto-optical disk or other storagedevice 103 to memory 102, and by executing the software performsprescribed functions.

The input/output device 104 is a keyboard, mouse or similar, whichreceives input from the host manager. The output device 105 is a displayor similar, which outputs information as instructed by the CPU 101. Theinterface control portion 106 is provided for connection to the IPnetwork 175, and the port 107 is provided for connection to the fibrechannel switch 120.

The management server 110 is a computer which manages operation andmaintenance of the entire computer system of this aspect, and is acomputer comprising a CPU 111, memory 112, storage device 113, inputdevice 114, and output device 115.

The input/output device 114 is a keyboard, mouse or similar, whichreceives input from the storage manager. The output device 115 is adisplay or similar, which outputs information as instructed by the CPU111. The interface control portion 116 is provided for connection to theIP network 175.

The CPU 111 reads storage management software and similar, stored on ahard disk, magneto-optical disk or other storage *device 113, intomemory 112, and by executing the software performs prescribed functions.

The management server 110 collects configuration information, resourceusage rates, performance monitoring information, fault logs and similarfrom various equipment within the computer system via the interfacecontrol portion 116 and IP network 175, according to the storagemanagement software, and outputs the collected information to the outputdevice 115, to present the information to the storage manager.

The management server 110 transmits operation and maintenanceinstructions, received from the storage manager via the input device114, to various equipment via the interface control portion 116.

The storage unit 130 is storage unit comprising external storageconnection functions, and further comprises one or more ports 131; oneor more control processors 132; one or more memory units 133 connectedto the control processors 132; one or more disk caches 134; one or morecontrol memory units 135; one or more ports 136; one or more diskdevices 137 connected to the ports 136; and an interface control portion138.

The control processor 132 identifies the device to be accessed for aninput/output request received from a host 131, and processes theinput/output request for a device within a disk device 137 or externalstorage unit 150 corresponding to the identified device.

The device to be accessed is identified by a port ID and LUN (LogicalUnit Number), contained within the input/output request received by acontrol processor 132.

In this aspect, the ports 131 are assumed to be ports to the fibrechannel interface which use SCSI (Small Computer System Interface) asthe higher-level protocol. However, the ports may also be ports to IPnetwork interfaces using SCSI as the higher-level protocol, or ports toother network interfaces for connection to storage unit.

The device to be accessed is identified from the port ID and LUNcontained in the input/output request as follows.

The storage unit 130 of this aspect has the following device hierarchy.

A disk array is configured from a plurality of disk devices 137. Thecontrol processors 132 manage the disk array as a physical device. Thecontrol processors 132 also allocate logical devices to the physicaldevices within the storage unit 130 (that is, the control processors 132associate physical devices with logical devices).

Here, logical devices are associated with LUNs allocated to each of theports 131, and are provided to hosts 100 as devices of the storage unit130. A logical device is managed within the storage unit 130, and itsnumber is managed independently for each storage unit. A host 100recognizes only logical devices of the storage unit 130. A host 100 usesthe LUN of a port 131 associated with a logical device to access datastored in the storage unit 130.

The storage unit 130 of this aspect also has functions to render virtualthe devices in external storage unit 150 as its own devices. A logicaldevice provided by external storage unit 150 (hereafter called an“external device”) to storage unit 130 is rendered virtual as a deviceof the storage unit 130 and provided to the host 100. Within the storageunit 130, an external device is, like a physical device within thestorage unit 130, associated with and managed as one or more logicaldevices of the storage unit 130.

In order to realize the above device hierarchy, a control processor 132manages the associative relations between logical devices, physicaldevices, disk devices 137, external devices, and the physical devices ofexternal storage unit 150. In this aspect, these associative relationsare retained in control memory 135.

A control processor 132 converts access requests for a logical deviceinto access requests for devices within a disk device 137 or for logicaldevices of external storage unit, based on the associative relationsmanaged by the control processor 132.

The storage unit 130 of this aspect combines a plurality of disk devices137 to define one or a plurality of physical devices (that is, aplurality of disk devices 137 are combined and associated as one or aplurality of physical devices), allocates one logical device to onephysical device, and provides this to a host 100. However, each diskdevice 137 may instead be provided to a host 100 as one physical deviceand as one logical device.

In addition to input/output processing for devices, a control processor132 also executes various processing to realize data links betweendevices, such as data copying and data redistribution.

Further, a control processor 132 transmits configuration information forpresentation to the storage manager to a management terminal 140,connected via the interface control portion 138; receives maintenanceand operation instructions, input by the manager to the managementterminal 140, from the management terminal 140; and alters theconfiguration of the storage unit 130 and similar according to thereceived instructions.

The above-described functions of control processors 132 are realizedthrough execution of a program stored in memory 133.

In order to improve the speed of processing of access requests from ahost 100, the disk cache 134 stores data which is frequently read fromthe disk devices 137, and also temporarily stores write data receivedfrom a host 100.

When performing write-after using the disk cache 134, in order toprevent loss of the write data stored in the disk cache 134 beforewriting to the disk device 137, it is desirable that the disk cache 134be made nonvolatile memory through battery backup or other means, orthat a duplicate configuration be employed to improve tolerance withrespect to media faults, or that other means be used to improve theavailability of the disk cache 134.

“Write-after” is processing in which, after write data received from ahost 100 is stored in the disk cache 134, and before actually writingthe data to the disk device 137, a response to the write request isreturned to the host 100.

The control memory 135 stores associative relations between devicesrealized in the above-described device hierarchy and attributes of eachdevice, as well as control information to manage these devices, andcontrol information in the disk cache 134 to manage data which eitherdoes or does not reflect disk data. If control information stored in thecontrol memory 135 disappears, data stored in a disk device 137 cannotbe accessed by a host 100, and so it is desirable that the controlmemory 135 be made nonvolatile memory through battery backup or othermeans, or that a duplicate configuration be employed to improvetolerance with respect to media faults, or that a configuration be usedto improve availability.

Each of the components in the storage unit 130 is connected by internalconnections as shown in FIG. 1. Through these internal connections,data, control information, and configuration information are transmittedand received between these components, and the control processors 132can share and manage configuration information for the storage unit 130.From the standpoint of improved availability, it is desirable that theinternal connections be made multiply redundant.

The management terminal 140 comprises a CPU 142; memory 143; storagedevice 144; interface control portion 141 connected to storage unit 130;interface control portion 147 connected to the IP network 175; inputdevice 145 which receives input from the storage manager; and outputdevice 146, such as a display or similar, which outputs to the storagemanager configuration information for storage unit 130 and managementinformation.

The CPU 142, by reading a storage management program stored in thestorage device 144 to memory 143 and executing the program, referencesconfiguration information, issues instructions to alter configurations,and issues instructions to execute specific functions.

The management terminal 140 serves as an interface, relating tomaintenance and operation of the storage unit 130, between the storagemanager or management server 110 and storage unit 130. The managementterminal 140 may be omitted, the storage unit 130 connected directly tothe management server 110, and the storage unit 130 managed usingmanagement software which runs on the management server 110.

Next, the software configuration of the storage unit 130 and managementserver 110 of this aspect is explained.

FIG. 2A is a software configuration diagram showing one example ofcontrol information stored in the control memory 135 and memory 133 ofthe storage unit 130, and of a program for storage control processing.

The control memory 135 stores logical device management information 201,physical device management information 202, external device managementinformation 203, LU path management information 204, and cachemanagement information 205. In this aspect, this control information isstored in control memory 135 in order to prevent information loss.

The control information stored in control memory 135 can be referencedand altered by a control processor 132. However, a control processor 132accesses control memory 135 via internal connections. In this aspect, inorder to improve processing performance, a copy of the controlinformation necessary for processing executed by each control processor132 is retained in memory 133 as a copy 211 of device managementinformation. The information retained as the copy 211 of devicemanagement information is the logical device management information 201,physical device management information 202, external device managementinformation 203, and LU path management information 204.

In addition to the copy 211 of device management information, the memory133 also stores an input/output request processing program 221, anexternal device monitoring processing program 222, and an externaldevice transfer processing program 223.

Device management information for the storage unit 130 is alsotransmitted to the control terminal 140 and management server 110, whereit is stored.

When the configuration of the storage unit 130 is altered by themanagement server 110 or management terminal 140 in conformance withstorage management software, or upon receiving an instruction from thestorage manager, or when the configuration of the storage unit 130changes due to a fault, automatic substitution or similar, one of thecontrol processors 132 updates the relevant device managementinformation in the control memory 135.

And, the control processor 132 which has updated the device managementinformation then notifies the other control processor 132, themanagement terminal 140, and the management server 110 of the fact thatthe relevant device management information has been updated.

FIG. 2B is a software configuration diagram showing one example ofcontrol information stored in the memory 112 of the management server110, as well as a program for storage control processing.

The memory 112 stores a copy 231 of device management informationcollected from the storage unit 130 and external storage unit 150, aswell as storage management information. 232 indicating the attributes ofthe storage unit 130 and external storage unit 150. In order to avoiddata loss, this information may also be retained in the storage device113 installed in the management server 110.

In addition, the memory 112 also stores a storage monitoring processingprogram 241 and an external device transfer instruction processingprogram 242.

Below, this control information is explained.

FIG. 3 shows one example of logical device management information 201.

Configuration information for each of the logical devices is stored inthe logical device management information 201. In this aspect, aninformation set comprising the logical device number 31, size 32,associated physical/external device number 33, device state 34, portnumber/target ID/LUN 35, connected host name 36, physical/externaldevice number during transfer 37, data transfer progress pointer 38, anddata transfer execution flag 39, is stored for each logical device inthe logical device management information 201.

A number uniquely allocated to each logical device by a controlprocessor 132 to identify the logical device is stored as the logicaldevice number 31.

The capacity of the logical device specified by the logical devicenumber 31 is stored as the size 32.

The number of the physical device or external device associated with thelogical device is stored as the associated physical/external devicenumber 33. In this aspect, the physical device number 51 or externaldevice number 61, which is stored in the physical device managementinformation 202 or external device management information 203 which aremanagement information for the device, is stored as the associatedphysical/external device number 33. Details of this are explained below.

In this aspect, logical devices and physical/external devices areassociated in a one-to-one correspondence. Consequently only one numberof an associated physical device or external device is stored as theassociated physical/external device number 33. When a plurality ofphysical/logical devices are combined to form a single logical device,an area becomes necessary in the logical device management information201 for storing a list of numbers of physical/external devicesassociated with each logical device, and the number of such numbers.Also, when a logical device is undefined, an invalid value is set as theassociated physical/external device number 33.

Information indicating the state of the logical device is set in thedevice state 34. States which may be set include “online”, “offline”,“uninstalled”, and “fault-offline”. “Online” indicates that the logicaldevice is operating normally and is in a state enabling access by a host100. “Offline” indicates that the logical device is defined and isoperating normally, but because the LU path is undefined or for someother reason, is not in a state enabling access by a host 100.“Uninstalled” indicates that the logical device is not defined, and sois not in a state enabling access by a host 100. “Fault-offline”indicates that a fault has occurred in the logical device, and thataccess by a host 100 is not possible.

The initial value of the device state 34 is “uninstalled”; when thelogical device is defined, this is changed to “offline”, and when the LUpath is defined, this is again changed to “online”.

The port number, target ID, and LUN are stored in the port number/targetID/LUN 35.

A port number stored in the entry 35 is information to identify a port131 of a logical device for which a LUN is defined. The portidentification information is a number, assigned to each port 131, whichis determined uniquely within the storage unit 130. Informationindicating to which port among the plurality of ports 131 the logicaldevice is connected, that is, the number of the port 131 used to accessthe logical device, is set in the entry 35.

The target ID and LUN stored in the entry 35 are identifiers used toidentify the logical device. In this aspect, as identifiers used toidentify a logical device, a SCSI-ID used for accessing by a host 100via SCSI, and the LUN, are stored.

The above-described values are set in the entry 35 when a LU pathdefinition is executed for a logical device.

The connection host name 36 is a host name which identifies the host 100which is permitted to access the logical device. As the host name, a WWN(World Wide Name) assigned to the port 107 of the host 100, or any othervalue capable of uniquely identifying the host 100 or the port 107, maybe used. The entry 36 is set by the storage manager at the time thelogical device is defined.

As the physical/external device number during transfer 37, thephysical/external device number of the transfer target of thephysical/external device to which the logical device is allocated duringdata transfer (when the data transfer execution flag 39, describedbelow, is “on”), is stored.

The data transfer progress pointer 38 is information indicating theleading address of the area for which data transfer processing has notbeen completed, and is updated as the data transfer progresses.

The initial value of the data transfer execution flag 39 is “off”, andwhen set to “on” indicates that data transfer is in progress, from thephysical/external device to which the logical device is allocated toanother physical/external device. The physical/external device numberduring transfer 37 and data transfer progress pointer 38 are valid onlywhen the data transfer execution flag 39 is set to “on”.

FIG. 4 shows an example of LU path management information 204. For eachof the ports 131 in the storage unit 130, the LU path managementinformation 204 stores information for a valid LUN defined for eachport.

A LUN defined for (allocated to) a port 131 is stored in the targetID/LUN 41. The number of the logical device to which the LUN isallocated is stored as the associated logical device number 42.Information indicating the host 100 allowed access to the LUN definedfor the port 131 is stored as the connected host name 43. The WWNassigned to the port 107 of the host 100 is for example used as theinformation indicating the host 100.

In some cases the LUNs of a plurality of ports 131 are defined for(allocated to) a single logical device, so that the logical device canbe accessed from a plurality of ports 131. In such cases, the union ofthe connected host names 43 of LU path management information 204 forall of the LUNs of the plurality of ports 131 is stored as the connectedhost name 36 of the logical device management information 201 for thelogical device.

FIG. 5 shows one example of physical device management information 202used for management of physical devices comprised by disk devices 137.

Each storage unit 130 retains for each physical device existing withinits equipment, as the physical device management information 202, aninformation set comprising the physical device number 51, size 52,associated logical device number 53, device state 54, RAID configuration(RAID level, data/parity disks) 55, stripe size 56, disk number list 57,start offset within disk 58, and size within disk 59.

An identification number to identify the physical device is registeredas the physical device number 51. The capacity of the physical devicespecified by the physical device number 51 is stored as the size 52. Thelogical device number associated with the physical device is stored asthe associated logical device number 53. The associated logical devicenumber 53 is stored at the time the logical device is defined. When thephysical device is not allocated to a logical device, an invalid valueis set as the associated logical device number 53.

Information indicating the state of the physical device is set in thedevice state 54. States which may be set include “online”, “offline”,“uninstalled”, and “fault-offline”. “Online” indicates that the physicaldevice is operating normally and is in a state of allocation to alogical device. “Offline” indicates that the physical device is definedand is operating normally, but is in a state of not being allocated to alogical device. “Uninstalled” indicates that the physical device is notdefined for the disk device 137. “Fault-offline” indicates that a faulthas occurred in the physical device, and that the physical device is notallocated to a logical device.

In this aspect, for simplicity it is assumed that physical devices arealready created in disk devices 137 at the time of factory shipment.Hence the initial value of device states 53 for physical devices whichcan be used is “offline”, and for other devices is “uninstalled”. At thetime that a logical device is defined for a physical device, the stateis changed to “online”.

Information relating to the RAID level, the number of data disks andparity disks, and other RAID configuration information for the diskdevice 137 to which a physical disk is allocated is stored in the RAIDconfiguration 55. The data division unit (stripe) length in the RAIDsystem is stored as the stripe size 56. Identification numbers for eachof the plurality of disk devices 137 comprised by the RAID system towhich the physical device is allocated are stored as the disk numberlist 57. The identification numbers for disk devices 137 are assignedvalues which are used to uniquely identify each disk device 137 in thestorage unit 130.

The start offset within disk 58 and size within disk 59 storeinformation indicating to which areas within the disk devices 137 aphysical device is allocated. In this aspect, for simplicity, it isassumed that, for all physical devices, the offset and size are unifiedwithin each disk device 137 comprised by the RAID system.

FIG. 6 shows one example of external device management information 203used to manage external devices provided to the storage unit 130 byexternal storage unit 150 connected to the storage unit 130.

For each external device, the storage unit 130 stores, as externaldevice management information 203, an external device number 61, size62, associated logical device number 63, device state 64, storageidentification information 65, external storage device number 66,initiator port number list 67, and target port ID/target ID/LUN list 68.

A value allocated uniquely within the storage unit 130 to the externaldevice by a control processor 132 is stored as the external devicenumber 61. The capacity of the external device specified by the externaldevice number 61 is stored as the size 62. The number of the logicaldevice within the storage unit 130 with which the external device isassociated is registered as the associated logical device number 63.

Information indicating the state of the external device is set as thedevice state 64. The states which can be set and their meanings are thesame as the device states 54 of the physical device managementinformation 202. Because the storage unit 130 is not connected to theexternal storage unit 150 in the initial state, the initial value of thedevice state 64 is “uninstalled”.

Information to identify the external storage unit 150 in which theexternal device is installed is saved as the storage identificationinformation 65. As identification information, a value may be used whichuniquely identifies the external storage unit 150. For example, acombination of vendor identification information and of a serial numberassigned uniquely by each vendor to the storage unit 150 may be used.

An identification number assigned to the external device by the externalstorage unit 150 in which the external device is installed is stored asthe external storage device number 66. In this aspect, an externaldevice is a logical device of external storage unit 150, and so thelogical device number assigned for use in identifying the logical devicewhich the external storage unit 150 itself has defined is stored as theexternal storage device number 66.

The identification number for a port 131 of storage unit 130 capable ofaccessing the external device is registered as the initiator port numberlist 67. When the external device can be accessed from a plurality ofports 131, all the identification numbers of ports capable of access areregistered.

When the external device defines LUNs for one or more ports of theexternal storage unit 150, one or a plurality of port IDs for theseports 151, and the target IDs/LUNs allocated to the external device, arestored as the target port ID/target ID/LUN list 68. When a controlprocessor 132 of the storage unit 130 accesses an external device (whenan input/output request is transmitted by the control processor from aport 131 to an external device), the target ID and LUN allocated to theexternal device by the external storage unit 150 to which the externaldevice belongs are used as information to identify the external device.

In this aspect, the storage unit 130 uses the above-described four itemsof device management information (logical device management information201, physical device management information 202, external devicemanagement information 203, and LU path management information 204) tomanage the device.

It is assumed that at the time of factory shipment of the storage unit130, physical devices are defined for each of the disk devices 137.Further, at the time of introduction of the storage unit 130 a user orstorage manager defines logical devices of external storage unit 150connected to the storage unit 130 as external devices, defines logicaldevices for the physical devices and external devices, and defines LUNsfor each port 131 for the defined logical devices.

FIG. 7 shows an example of storage management information 232 in themanagement server 110.

Information used to manage the storage unit 130 and external storageunit 150 managed by the management server 110 is stored in the storagemanagement information 232. In the following explanation of the storagemanagement information 232, when there is no need in particular todistinguish the storage unit 130 and external storage unit 150, both arerepresented as “storage unit”. Similarly, disk devices 137, 156 andcontrol processors 132, 152 which are components of storage unit arerepresented as “disk devices” and “control processors”.

An information set comprising, for each storage unit, a storage number71, storage name 72, port name list 73, performance/reliability level74, total capacity 75, and free capacity 76, is stored as the storagemanagement information 232.

A number determined uniquely within the system and allocated to eachstorage unit by the management server 110 is stored as the storagenumber 71.

Information indicating an identifier used to specify the storage unit isregistered as the storage name 72. As the identifier, the platform WWNof the fibre channel, or a combination of the vendor identifier andproduct number for the storage unit, may be used.

WWNs assigned to ports of the storage unit are stored in the port namelist 73. A host 100 uses the port WWNs of the storage unit 130 stored inthe port name list 73 to specify a port to be used when accessing adevice in the storage unit 130.

Values representing evaluations, based on unified standards for computersystems, of the performance and reliability of the storage unit, arestored in the performance/reliability level 74.

Indexes used to evaluate performance may include such performance valuesas the seek time and disk rotation speed of the disk devices installedin the storage unit, the storage capacities of disk devices, the RAIDlevel configuration in the storage unit, the communication bandwidth ofconnections between control processors and disk devices, portcommunication bandwidths, the number of communication lines, the storagecapacity of the disk cache, and nominal performance values for thestorage unit overall.

Depending on the storage unit, there are cases in which disk deviceswith different attributes and RAID configurations with differentattributes coexist within the equipment, so that there are a pluralityof performance levels within a single storage unit. But in this aspect,for simplicity, it is assumed that the performance level is set for eachstorage unit, and can be managed for each storage unit.

Indexes used to evaluate reliability may include the redundancy of thedisk devices, control processors, or other components of the storageunit, the RAID level used by the storage unit, the number ofsubstitution paths which can be used, and various other conditionsrelated to product specifications. The various functions of the storageunit, such as for example functions provided by the storage unit forcopying or saving logical devices, can also be used as indexes inevaluating reliability.

With respect to the reliability level also, depending on the storageunit it is possible for storage areas with different reliability levelsto coexist internally; but to simplify the explanation, in this aspectit is assumed that each storage unit has a single reliability level, andthat each storage unit can be managed individually.

In this aspect, performance and reliability levels are managed usingfive stages of values, from a maximum of “5” to a minimum of “1”. Thevalue of the level for each storage unit is determined and set by thestorage manager based on catalog values for the storage unit and on theresults of tests at the time of equipment introduction.

Information indicating the total capacity of storage areas which can beused in the storage unit is registered as the total capacity 75. Thetotal capacity of storage area which can be used is determined by thestorage capacities of disk devices in the storage unit, and by the RAIDlevel configuration in the storage unit. In this aspect, it is assumedthat physical devices which can be used are set in advance, and that thetotal capacity of physical devices which can be used is registered asthe total capacity 75.

Information indicating the total capacity of physical devices for whicha logical device is not yet defined, among all the physical devices inthe storage unit, is registered as the free capacity 76. In this aspect,information indicating the total storage capacity of physical devices inthe “offline” state is registered as the free capacity 76. Becausephysical devices in the “uninstalled” state cannot be used by a host100, the capacity of such devices is not included.

In the case of an aspect in which a physical device required by themanagement server 110 is defined according to instructions from a useror storage manager, information indicating the total capacity of unusedareas in disk devices installed in the storage unit is registered as thefree capacity 76.

Next, returning to FIG. 2, programs stored in the memory 133 and 112 ofthe storage unit 130 and management server 110 are explained. Theseprograms are executed by each of the control processors and CPUs.

The input/output request processing program 221, external devicemonitoring processing program 222, and external device transferprocessing program 223, which are stored in memory 133 of the storageunit 130, as well as the storage monitoring processing program 241 andexternal device transfer instruction processing program 242, which arestored in memory 112 of the management server 110, are explained.

The input/output request processing program 221 realizes input/outputprocessing for a logical device. Upon detecting an external deviceanomaly (a phenomenon which is a prognostication of the occurrence of afault in an external device) during input/output processing, theinput/output request processing program 221 notifies the managementserver 110.

The external device monitoring processing program 222 periodicallymonitors external devices, and upon detecting an anomaly in an externaldevice, notifies the management server 110.

The external device transfer processing program 223 performs processingto transfer the data of a specified external device to another device,according to an instruction from the management server 110.

The storage monitoring processing program 241 receives warnings ofanomalies in external devices and fault reports from the storage unit130 and external storage unit 150, creates transfer plans for externaldevices according to received reports and similar, and issuesinstructions for transfer of external device data to the storage unit130.

The external device transfer instruction processing program 242determines the transfer target when an external device for transfer isspecified by the storage monitoring processing program 241.

These programs are used in storage control processing within the variouscomponents as explained below.

Data transfer instructions issued when an external device anomaly isdetected are executed in concert by the input/output request processingprogram 221 and/or external device monitoring processing program 222 ofthe storage unit 130, and by the storage monitoring processing program241 of the management server 110.

Processing to detect anomalies in external devices during input/outputrequest processing, which is performed by the input/output requestprocessing program 221, is explained below.

FIG. 8 shows an example of the flow of processing to detect anomalies inexternal devices during input/output request processing, performed bythe input/output request processing program 221.

A control processor 132 identifies the physical device or externaldevice associated with the logical device of an input/output requestreceived, from a host 100 at each port 131, for a logical device of thestorage unit 130, and performs input/output processing for the physicaldevice, or transmits an input/output request to the external storageunit 150 of the external device, according to the input/output requestprocessing program 221.

In this aspect, upon receiving a fibre channel command frame (step 801),the control processor 132 references the LU path management information204 and logical device management information 201, and acquires thelogical device number which the frame is to access from the LUNcontained in the received frame, as well as the physical device numberor external device number associated with the logical device (step 802).

When the acquired logical device is associated with a physical device inthe storage unit 130, the control processor 132 performs datainput/output processing for the disk device 137 housing the physicaldevice, using the disk cache 134, to complete the input/output requestprocessing (step 803).

When on the other hand the logical device is an external device, thecontrol processor 132 performs input/output processing for the externaldevice via a port 131 (step 804). An input/output request for anexternal device entails essentially the same processing as aninput/output request issued by a host 100 for a logical device presentedby the storage unit 130.

If, during input/output processing for an external device, an accessfault, decline in performance, or other external device anomaly isdetected (step 805), the control processor 132 warns the managementserver 110 of the detection of an external device anomaly (step 806).The warning should include information enabling identification of thefact that an access fault or performance fault has occurred. A warningmay also include information to identify the external device forinput/output processing, information indicating the grounds for judgingan anomaly to have occurred, and similar.

If an external device anomaly is not detected, normal input/outputprocessing is performed.

Detection of access faults or performance decreases in this aspect isperformed as follows.

Access faults are judged and detected through responses to input/outputrequests which have been sent.

External device access faults occur when, for example, a fault (due tocutting or removal of a cable, a switch fault, or similar) occurs in thenetwork leading from ports 131 of the storage unit 130 to ports 151 ofthe external storage unit 150, or when a fault occurs in a port 151 ofthe external storage unit 150, in a control processor 152, or similar.

Such access faults are detected through time-outs, as seen by thecontrol processor 132, of input/output requests transmitted to anexternal device, because access through the specified port 151 is notpossible. Having detected the time-out of an input/output request, acontrol processor 132 executes substitution path processing, similarlyto normal cases for disk devices within the storage unit.

First, when an input/output request using a specified port 151 timesout, the control processor 132 confirms the state of the path to theexternal storage unit 150 using the port in question 151.

If the path state is normal, a specified number of input/output requestsare again sent over the same path, and if not all of these aresuccessful, the port is switched to a substitute port, and input/outputrequests are sent once again. If the path state is not normal, repeatedtrials of the path are skipped, and switching to a substitute port isperformed first before resending input/output requests.

If input/output requests from the substitute port are processed withoutincident, the control processor 132 transmits to the management server110 a warning message indicating the fact of occurrence of an accessfault in the external device.

When, as a result of the above processing, notification of a change inthe network state is received, if as a result of checks of the links forall ports 151 of external storage unit 150 for which links have beenestablished (in a fibre channel, node port login) it is found that alink is broken, or when in input/output processing for an externaldevice time-outs have occurred more than a specified number of times fora path using a specified port 151, then the path state is changed to a“blocked” state.

In this aspect, when input/output requests fail for all substituteports, input to and output from the relevant external device is notpossible, and data is lost.

Performance decreases are detected through decreases in responsivenessand throughput of input/output requests for an external device. Eachtime an input/output request is sent, the control processor 132 acquiresthe response time and throughput information for the request. Theaverage response times and throughput values acquired in advance arecompared for each external storage unit 150, and when the divergencebetween values is large, an anomaly is judged to have occurred. Thedivergence threshold value for judgment of occurrence of an anomaly isstored in for example the memory 133 of the storage unit 130, togetherwith information on average response times and throughput.

When, because there is divergence in responsiveness and throughput, itis judged that an anomaly has occurred, information indicating thegrounds for this judgment (the responsiveness or throughput) is includedin the warning.

Degradation of responsiveness or throughput may occur, for example, as aresult of such anomalies as single-sided blockage of the disk cache 154.In normal write processing, a completion response is sent when duplicatewriting to the disk cache 154 of the external storage unit 150 iscompleted. But when there is blockage of one of the disk caches 1.54 towhich duplicate writing of data is performed, write-through occurs inwhich the completion response is sent only when direct writing to thedisk device 156 is completed. In write-through mode, the writeperformance drops dramatically, and problems such as degradation ofresponsiveness and throughput occur.

The detection of an access fault or performance decline signifies adecline in the redundancy of the network, processor, or similar whichguarantees access to the external device. Hence in order to guaranteeaccess to data stored in the external device, either redundancy must berestored quickly, or the data of the external device must be saved to(transferred to) another device.

A control processor 132 which has detected an anomaly transmits amessage or signal to the management server 110 warning of an anomaly inthe external device, and causes the management server 110 to acknowledgethe occurrence of the anomaly.

Next, processing to detect anomalies in external devices by the externaldevice monitoring processing program 222 is explained. A controlprocessor 132 periodically monitors the operating state of externaldevices according to the external device monitoring processing program222.

The occurrence of faults during input/output processing in an externaldevice which is accessed by a host 100 with a certain frequency can bedetected according to the input/output request processing program 221.However, in the case of external devices storing archive data, or in thecases of other devices accessing of which occurs only rarely, it isnecessary to monitor the state of the external device on occasions otherthan accessing by a host 100. Consequently external device monitoringprocessing is provided, according to the external device monitoringprocessing program 222.

FIG. 9 is one example of the flow of processing to detect anomalies inan external device by the external device monitoring processing program222.

The control processor 132 periodically starts the external devicemonitoring processing program 222 with a predetermined frequency. Thestartup frequency is set so as not to impede input/output requests fromhosts 100.

The control processor 132 selects the external device for which toperform trial input/output from among all the external devices beingmanaged and described in the external device management information 203(step 901), and executes test I/O (for example, read processing) for theexternal device thus selected (step 902), according to the externaldevice monitoring processing program 222.

In this step, the external device for testing is selected each timebased on the time elapsed from the last time the device was accessed bya host 100, and other criteria. The method of selection is not limitedto this method. Further, in this aspect trial input/output is performedfor one external device upon each startup; but trial input/output may beperformed for a plurality of external devices.

In the trial input/output for the selected external device, when ananomaly is detected in the external device (step 903), the controlprocessor 132 warns the management server 110 of the external deviceanomaly (step 904). The method of anomaly detection is the same as instep 805 of the flow of input/output request processing, and so anexplanation is omitted.

In this way, when the control processor 132 detects an anomaly which mayimpede access to an external device being managed, it warns themanagement server 110 of this fact. In the management server 110,transfer processing of the external device is performed, based on theexternal device anomaly warning from the storage unit 130, which isvirtualized storage, and/or on a fault occurrence report from storageunit being managed (the external storage unit 150 which is virtualstorage).

Below is an explanation of the processing performed by the managementserver 110 upon receiving a warning from the storage unit 130 indicatingthe occurrence of an anomaly in an external device (hereafter called“anomaly warnings”), and/or a fault occurrence report from storage unitbeing managed (external storage unit 150).

FIG. 10 is one example of the flow of processing of the storagemonitoring processing program 241 executed by the management server 110.The CPU 111 performs the following processing by executing the storagemonitoring processing program 241.

The CPU 111 receives an anomaly warning from the storage unit 130,and/or a fault occurrence report from the external storage unit 150(step 1001).

The CPU 111 analyzes the received anomaly warning and/or fault report(step 1002).

The CPU 111 decides which external devices are affected, according toinformation stores in the received anomaly warning and/or fault report,and also judges whether data transfer is necessary for logical devicesjudged to be affected, and selects the range of external devices(logical device group) for transfer (step 1003).

Here, upon receiving an anomaly warning, the CPU 111 extractsinformation for the external device stored in the anomaly warning aswell as the anomaly details (access fault, decline in responsiveness orthroughput). Based on the extracted external device information andanomaly details, the external storage unit 150 comprising the externaldevice is accessed, and existing techniques are used to investigate thedetails of the location of fault occurrence, the extent of the fault,and similar.

When a fault report is received, the information stored in the faultreport is used to identify the location of fault occurrence, extent ofthe fault, and similar.

The location of fault occurrence is for example the site of the fan,power supply, disk cache, port, disk device, or similar of the externalstorage unit 150 for which a fault has been reported; the extent of thefault is a level indicating whether, due to the fault occurrence, thesite cannot be used, or whether the fault is temporary and recovery tonormal is already in progress with respect to configuration informationof the storage unit; the extent of the fault can be judged frominformation on the type of fault. In the latter case, recovery to thenormal state is in progress, and so no action need be taken with respectto the external device in question.

The CPU 111 receives the latest configuration information, includingdevice information, from the external storage unit 150 for which ananomaly warning and/or fault report was issued, and identifies thelogical device group for which availability is reduced as a consequenceof the fault.

For example, when the site of the fault occurrence is the fan and powersupply, and if there are few remaining replacements for the fan andpower supply, the availability of all logical devices mounted in thestorage unit is reduced. In this case, the CPU 111 determines that therange of reduced availability is the entirety of logical devices.

When a fault occurs in one side of the doubly redundant memory of thedisk cache 154 due to a fault, it is anticipated that there will be asharp decline in the availability and performance level of all thelogical devices of the external storage unit 150 for which the anomalywarning and/or fault report is issued. In this case also, the CPU 111determines that the range over which availability is degraded extends toall the logical devices of the external storage unit 150.

When a fault occurs in a specific disk device 156, and the redundancy ofthe RAID group to which the disk device 156 belongs is lost, and ifthere remain no substitute disk devices within the external storage unit150 comprising the disk device 156, then the availability of the logicaldisk group associated with the RAID group is reduced. In this case, theCPU 111 determines that the range over which availability is degraded isthe logical device group associated with the RAID group to which thedisk device 156 in which the fault has occurred belongs.

When the logical device group which is affected has been determined, theCPU 111 uses the external device management information within the copy231 of the device management information to investigate whether, in thelogical device group affected by the reported fault, there exist anydevices which are managed as the external devices of other storage unit.

When the external device group for transfer is determined, the CPU 111issues an instruction for transfer of the data within the externaldevice for transfer to the storage unit 130, according to the externaldevice transfer instruction processing program 242 (step 1004).

When external device transfer (data transfer) by the storage unit 130 iscompleted, and a transfer completed notification is received from thestorage unit 130, the CPU 111 receives into memory 112 the updateddevice configuration information for the storage unit 130 (logicaldevice management information 201, physical device managementinformation 202, external device management information 203, LU pathmanagement information 204) as a copy 231 of the device managementinformation, and processing is concluded (step 1005).

Next, details of the processing of the above step 1004, performedaccording to the external device transfer instruction processing program242, are explained.

FIG. 11 is one example of the flow of processing by the CPU 111 of themanagement server 110, according to the external device transferinstruction processing program 242.

When external devices for which device transfer is necessary aredetermined according to the storage monitoring processing program 241,the CPU 111 takes these external devices to be the transfer source,determines the transfer target device, and issues an external devicetransfer instruction (instruction to perform data transfer) to thestorage unit 130.

First, the CPU 111 references the storage management information 232 andsimilar, to confirm the performance, reliability level, and otherattributes of the transfer source devices (step 1101).

The CPU 111 investigates whether there exists an unused physical devicein the external storage unit 150a, 150b under management by the storageunit 130 which has virtualized and managed the external devices whichare the transfer source devices or in the storage unit 130, that is, a(free) device not allocated to a logical device and having aperformance/reliability level and similar equal to or exceeding that ofthe transfer source devices (step 1102).

The copy 231 of device management information for each of the storageunits and the storage management information 232 in the memory 112 ofthe management server 110 are used in this investigation of freedevices.

When there exists a free device under the management of the storage unit130, which is unused and satisfies the above conditions, the CPU 111determines this device to be the data transfer target (step 1105). Whenthe transfer target is determined, the CPU 111 transmits an externaldevice transfer instruction to the storage unit 130 (step 1106).Information specifying the transfer source and transfer target iscontained in the external device transfer instruction. In this aspect,the external device number 61 of the external devices is used. In casesin which the transfer target is a device of the storage unit 130, thephysical device number 51 is used instead of an external device number61.

When on the other hand no free device exists, the CPU 111 investigateswhether there exists a free device satisfying the conditions withinstorage unit which is under the management of the management server 110,and which is not under the virtualized control of the storage unit 130(step 1103).

If a free device satisfying the conditions is found, the CPU 111instructs the storage unit 130 to register the device as an externaldevice (step 1104).

The processing performed in step 1104 is similar to the processing,performed at the time of system construction, in which the logicaldevices of other external storage unit 150 connected to the storage unit130 are registered as external devices.

Specifically, the control processor 132 issues an inquiry to theexternal storage unit 150 in question, and registers the external devicemanagement information 203. Then, in the management server 110, thelogical devices of the external storage unit in question are associatedas external devices of the storage unit 130, and the copy 231 of thedevice management information is updated.

The CPU 111 selects an external device registered in step 1104 as thetransfer target (step 1105), and issues an external device transferinstruction to the storage unit 130 (step 1106).

On the other hand, when in the investigation of step 1103 a free devicesatisfying the conditions is not found, an investigation of theexistence of devices is performed once again within the range ofinvestigation of step 1102, that is, free devices under the virtualizedcontrol of the storage unit 130 which, though not satisfying theperformance/reliability level condition, have the capacity of thetransfer source devices (step 1107). This is done in order to avoidstoring data in a device in which a fault has been discovered.

When a free device which satisfies only the capacity condition isdiscovered, the CPU 111 selects this device as the transfer target (step1105), and issues an external device transfer instruction to the storageunit 130 (step 1106).

When a free device is not found, an investigation of the existence ofdevices is performed once again within the range of investigation of thenext step 1103, that is, free devices under the management of themanagement server 110 and not under the virtualized control of thestorage unit 130 which, though not satisfying theperformance/reliability level condition, satisfy the capacity condition(step 1108).

When a free device is discovered, the CPU 111 instructs the storage unit130 to register the free device as an external device (step 1104),selects the device as the transfer target (step 1105), and issues anexternal device transfer instruction to the storage unit 130 (step1106).

When a free device cannot be found in step 1108 either, an output device115 or similar means are used to inform the storage manager of the factthat transfer of the transfer source external device is not possible,and processing is interrupted (step 1109).

Next, processing of the storage unit 130 upon receiving an externaldevice transfer instruction from the management server 110 is explained.

FIG. 12 shows one example of the flow of external device transferprocessing, executed by a control processor 132 according to theexternal device transfer processing program 223.

External device transfer processing is processing to transfer the dataof a transfer source external device specified by the management server110 to a transfer target device (an external device, or a physicaldevice of the storage unit 130).

Upon receiving an external device transfer instruction from themanagement server 110, the control processor 132 registers the devicetransfer state in the logical device management information 201 for thelogical device associated with the transfer source external device (step1201).

Here, the control processor 132 sets the external device number 61 orphysical device number 51 which is the transfer target in thephysical/external device number during transfer 37, initializes the datatransfer progress pointer 38 to 0, and sets the data transfer executionflag 39 to “On”.

The control processor 132 then executes sequential data transfer fromthe transfer source external device to the transfer targetphysical/external device, from the beginning to the end, according tothe external device transfer processing program 223, and in accordancewith the data transfer progress pointer 38 (step 1202).

In this aspect, the control processor 132 executes this external devicetransfer processing while receiving input/output from hosts 100. Whenduring data transfer there is an input/output request from a host 100for a logical device associated with an external device which is thetransfer source, the control processor 132 uses the data transferprogress pointer 38 of the logical device management information 201 tojudge whether transfer of the data to be accessed has been completed. Inthe case of input/output for areas the transfer processing of which isjudged not to have been completed, duplicate writing to both the areasof the transfer source and transfer target devices, and similar controlis executed.

When data transfer up to the end of the transfer source external deviceis completed, the control processor 132 updates the logical devicemanagement information 201, external device management information 203,and physical device management information 202 for the logical devices,physical devices, and external devices involved in the data transfer(step 1203).

That is, the associative relation between logical devices and externaldevices or physical devices after the completion of data transfer isstored in these types of management information.

Here, the external/physical device number of the transfer target is setin the associated physical/external device number 33 of the logicaldevice management information 201, and the data transfer execution flag39 is set to “Off”.

When the transfer target is a physical device, the number of the logicaldevice set as the transfer target external/physical device number in theassociated physical/external device number 33 is set in the associatedlogical device number 53 of the physical device management information202, and the device state 54 is set to “online”.

When the transfer target is an external device, the number of thelogical device set as the transfer target external/physical devicenumber in the associated physical/external device number 33 is set inthe associated logical device number 63 of the external devicemanagement information 203, and the device state 64 is set to “online”.

Further, an invalid value is set as the associated logical device number63 of the external device management information 203 for the transfersource external device, and the device state 64 is set to “offline”.

When updating of the different types of management information iscompleted, the control processor 132 notifies the management server ofthe fact that external device transfer processing has been completed(step 1204).

As explained above, in this aspect appropriate control can be executedin storage having external storage connection functions, enabling thedetection of anomalies which are prognostications of faults occurring inexternal storage devices, the identification of the range of equipmentaffected, and the execution of data transfer.

Hence in this aspect, even when, in a computer system having a storagesystem comprising storage unit having external storage connectionfunctions and external storage unit connected to the above storage unit,the external storage unit is a storage device with comparatively lowreliability, the availability of the system as a whole can be improved.

When new storage unit is introduced and the overall capacity of thestorage system is increased, and all the data held by devices inexisting storage is transferred to the new storage unit to replace theabove, it is necessary that the newly introduced storage unit comprisecapacity equal to that of the existing storage devices, and so the costof storage unit introduction is increased. Further, if existing storageunit and new storage unit are both connected directly to hosts, controlon the host side becomes complicated.

By means of this aspect, new storage unit can be introduced withoutmodifying the mode of access by hosts, and a computer system can beconstructed in which existing storage unit can be effectively utilized.As a result, the cost of equipment introduction can be reduced.

Second Aspect

Next, a second aspect is explained. Here, only differences with thefirst aspect are explained.

The hardware configuration of a computer system to which this aspect isapplied is similar to that of the first aspect, shown in FIG. 1. In thisaspect, a plurality of loigcal devices of the one or more externalstorage units 150 shown in the figure are collected to constitute a RAIDgroup. In this aspect, logical devices are defined for this RAID group.For simplicity, a one-to-one correspondence between logical devices andRAID groups is assumed.

By forming RAID groups from the logical devices of the external storageunit 150, in this aspect two types of methods to protect the data ofexternal devices are possible; these are the data transfer methodexplained in the first aspect, and a data recovery method using anotherdevice comprised by the RAID group after an external device can nolonger be accessed.

In this aspect, because RAID groups are configured from logical devicesof external storage unit 150, the data stored as logical devicemanagement data 201 differs from that in the first aspect.

FIG. 14 shows an example of the configuration of logical devicemanagement information 201.

The logical device management information 201 in this aspect comprises,for each logical device, a logical device number 1401; size 1402;associated physical/external device number 1403; device state 1404; RAIDconfiguration (RAID level, data/parity disks) 1405; stripe size 1406;physical/external device number list 1407; transfer/recovery sourcephysical/logical device number and transfer/recovery targetphysical/logical device number 1408; data transfer/recovery progresspointer 1409; and data transfer/recovery execution flag 1410.

An identification number to identify the logical device is registered asthe logical device number 1401. The capacity of the logical devicespecified by the logical device number 1401 is stored as the size 1402.

The physical/external device number associated with the logical deviceis stored as the associated physical/external device number 1403. Thisnumber is stored at the time of definition of the logical device. Whenthe logical device is not allocated to a physical/external device, aninvalid value is set as the associated physical/external device number1403.

Similarly to the first aspect, information indicating the state of thelogical device is set as the device state 1404.

Information relating to the RAID level, number of data disks, number ofparity disks and similar of the RAID group constituting thephysical/external device allocated to the logical device is stored inthe RAID configuration 1405. Similarly, the data division unit (stripe)length in the RAID group is stored in the stripe size 1406.Identification numbers for each of the plurality of physical/externaldevices comprised by the RAID group to which the logical device isallocated are stored in the physical/external device number list 1407.

In this aspect, because data recovery processing is also performed, thedata transfer/recovery execution flag 1410 can assume three values,which are “data being transferred”, “data being recovered”, and “off”.When the data of an external device is transferred to anotherphysical/external device, the flag is set to “data being transferred”,and when data recovery is being performed for an external device whichcan no longer be accessed, the flag is set to “data being recovered”.

When, because of a fault or some other reason, data is transferred toanother external/physical device or data recovery is being performed,the external/physical device number for the transfer/recovery source,and the external/physical device number for the transfer/recoverytarget, are set in the transfer/recovery source physical/logical devicenumber and transfer/recovery target physical/logical device number 1408.Information indicating the leading address of the area for which datatransfer or recovery processing has not been completed is stored in thedata transfer/recovery progress pointer 1409. This value is updated asthe data transfer or recovery processing advances.

At the time of initiation of data transfer/recovery processing, theentry 1408 is set, the entry 1409 is initialized, and the entry 1410 isset to a value indicating “data being recovered” or “data beingtransferred”.

In this aspect, the storage unit 130 manages the device hierarchy usingfour types of device management information, similarly to the firstaspect. Logical devices are defined by combining pluralities of physicaldevices defined in advance, and external devices defined by users or bystorage managers. LUNs are defined for each of the ports 131 of logicaldevices defined in this way.

In this aspect, a plurality of external devices and physical devices arecombined to configure a RAID group. Similarly to the first aspect,decreases in the availability of components supporting access toexternal devices are detected, external device faults are predicted, andpreventative data transfer is performed in advance; in addition, aftereither an external device or a physical device has become inaccessible,the information of the external device which has become inaccessible canbe recovered from other physical/external devices comprised by the RAIDgroup.

The fact that a physical/external device has become inaccessible can bedetected by processing to detect external device anomalies according tothe external device monitoring processing program 222 and input/outputrequest processing program 221, which were also comprised by the firstaspect. Detection is also possible through fault reports from externalstorage unit 150, obtained from the existing functions of the managementserver.

In this aspect, the storage unit 130 comprises an external devicerecovery processing program (not shown) in memory 133, and themanagement server 110 comprises an external device recovery instructionprocessing program (not shown) in memory 112, in order to performrecovery processing for an external device or physical device which hasbecome inaccessible and has been detected by the above-describedprograms.

The external device recovery processing program is executed by a controlprocessor 132, and the external device recovery instruction processingprogram is executed by the CPU 111, to perform their respectivefunctions.

The functions realized by the input/output request processing program221, external device monitoring processing program 222, and storagemonitoring processing program 241 are similar to those in the firstaspect, and so an explanation is here omitted.

The management server 110 of this aspect selects the recovery targetdevice (physical/external device) to store recovery data according tothe external device recovery instruction processing program, and issuesa data recovery instruction to the storage unit 130. The flow ofprocessing of the external device recovery instruction processingprogram is similar to the flow of processing of the external devicetransfer instruction processing program 242 explained in the firstaspect, but with “transfer instruction” replaced with “recoveryinstruction”, and “transfer source/target device” replaced with“recovery source/target device”, and so an explanation is here omitted.

FIG. 13 shows an example of the flow of processing for external devicerecovery in this aspect, according to the external device recoveryprocessing program executed by a control processor 132.

Upon receiving an external device recovery instruction from themanagement server 110, the control processor 132 executes externaldevice recovery processing according to the external device recoveryprocessing program.

The control processor 132 registers information indicating that recoveryis being executed in the management information for the logical devicefor which the recovery instruction was received (step 1301). At thistime, the logical device management information 201 for this logicaldevice, the external device management information 203 for the recoverysource and recovery target, and the physical device managementinformation 202 for the recovery source and recovery target are updated,and the logical device association is replaced.

The control processor 132 recovers the data from anotherexternal/physical device for which the logical device, to which therecovery source external device belongs, is defined, and stores the datain the recovery target external/physical device, in sequence from theleading address (step 1302).

When data recovery up to the end of the device is completed, the controlprocessor 132 sets the data transfer/recovery execution flag 1410indicating the processing state of the logical device to “off” (step1303), and notifies the management server 110 of the completion ofrecovery processing (step 1304).

As explained above, by means of this aspect, not only can an externaldevice fault be predicted and data within the device be transferred as apreventive measure, similarly to the first aspect, but after a fault hasactually occurred and access is no longer possible, the data within theexternal device can be recovered.

Hence through the configuration of this aspect, in a computer systemwhich presents to hosts in virtualized form the devices within firststorage unit connected to second storage unit as its own devices, faultmanagement and handling can be realized for the devices of the firststorage unit, so that the availability of the computer system as a wholecan be improved.

This invention is not limited to the above-described two aspects, butcan be variously modified.

In the above two aspects, the management server 110 determines whetherexternal device transfer should be performed, based onperformance/reliability level information maintained for each storageunit, the site of the fault occurrence, and the level of the fault whichhas occurred.

However, in addition to the above decision criteria, a user or storagemanager may define the reliability level required for each logicaldevice in advance, and measures to cope with external device faults maybe decided according to such reliability levels.

For example, whether or not to perform device transfer for an externaldevice anticipated to become inaccessible due to a fault may be decidedbased on the (required) reliability level of the logical device definedfor the physical device. In other words, when the required reliabilitylevel is high, processing is performed immediately for transfer toanother external/physical device, but when the required reliabilitylevel is low, transfer processing is not performed, a fault report issent to the storage manager, and an instruction from the storage manageris awaited.

In the above two aspects, the transfer target/recovery target device isdetermined by giving priority to selection of external devices or ofphysical devices within the storage unit 130 which satisfy theperformance/reliability level requirements defined for the externaldevice for which an anomaly has been detected. However, reduction of thetime required for transfer completion may be emphasized to select a freedevice within the storage unit 130 meeting only the capacity condition,and after data transfer is completed the data may be transferred toanother device meeting the performance/reliability level condition.

In the second aspect, an example was explained in which, when one of theexternal devices comprised by a RAID group becomes inaccessible, onlythe data within the external device which has become inaccessible isrecovered to another physical/external device.

However, when another device comprised by the RAID group is in the sameexternal storage unit 150 as the external device which has becomeinaccessible, it is possible that a plurality of other devices in theRAID group may also be subjected to data transfer. In such cases, theexternal device transfer instruction processing program 242 selects arecovery target device for the external device in which the fault hasoccurred, and selects a transfer target device for external devicesother than the external device of the fault and which are to besubjected to data transfer, and issues a device recovery instruction andtransfer instruction to the storage unit 130. The storage unit 130 readsthe data of the physical/external devices comprising the RAID group, andperforms processing in parallel to both recover data to the recoverytarget device and also to transfer the data of the external devices forread transfer to transfer target devices.

1. A management server, managing a second storage system which providesa host computer with access to both a second physical device in thesecond storage system and a first logical device in a first storagesystem coupled to the second storage system, comprising: transfer sourcedecision means for identifying, based on information received from saidsecond storage system prognosticating a fault in said first logicaldevice, a storage area affected by said fault as the transfer source;and, data transfer instruction means for selecting, based on the datacapacity and evaluation of the performance and reliability levelsdetermined in advance of said transfer source, the storage area of atransfer target from among the storage areas of said first storagesystem and said second storage system managed by said second storagesystem, and for issuing to said second storage system an instruction totransfer the data of said transfer source to said transfer target. 2.The management server according to claim 1, wherein said data transferinstruction means selects a device, satisfying both the conditions ofhaving the data capacity of said transfer source and of being evaluatedas meeting said predetermined performance and reliability levels, fromamong the storage areas of said first storage system or of said secondstorage system managed by said second storage system, and determinessaid device to be the transfer target storage area; when no deviceexists which satisfies both of said conditions within the storage areasof said first storage system or said second storage system managed bysaid second storage system, a device which at least satisfies said datacapacity condition is selected, and said device is determined to be thetransfer source storage area; when no device exists which satisfies atleast said data capacity condition within storage areas of said firststorage system or said second storage system managed by said secondstorage system, a device is selected which satisfies both said datacapacity condition and said performance and reliability level evaluationcondition from among the storage area of another storage system beingmanaged by said management server, said device is determined as thetransfer target storage area, and an instruction is issued to saidsecond storage system to register the device selected as the transfertarget-as an external device providing access to the host computer; whenno device exists which satisfies both of said conditions within thestorage areas of said other storage systems, a device which satisfies atleast said data capacity condition is selected from among the storageareas of said other storage systems, the device is determined as thetransfer target, and an instruction is issued to said second storagesystem to register the device selected as the transfer target as anexternal device of said second storage system; and, when no deviceexists which satisfies at least said data capacity condition within thestorage areas of said other storage systems, information indicating thatdata transfer is not possible is output.
 3. A storage system, managingas logical devices both external devices in an external storage systemcoupled to said storage system and a physical device in said storagesystem, comprising: logical device management means for managing bothphysical devices installed within said storage system itself and saidexternal devices as device configuration information in association withlogical devices provided to a host; input/output processing means forperforming processing, when an input/output processing request for alogical device is received from said host, to convert the receivedinput/output request into input/output processing for said physicaldevice or said external device based on said device configurationinformation, and to perform input/output processing according to saidconversion result; anomaly detection means for judging the occurrence,during input/output processing for an external device by saidinput/output processing means, of an access fault or performance declinefor the external device; notification means for issuing, when an accessfault or decline in performance has been detected by said anomalydetection means, notification of the detected anomaly and the accessedexternal device to the management server which manages said storagesystem and said external storage system; and, data transfer means forperforming, upon receipt of an instruction from said management serverfor data transfer specifying the transfer source and transfer target,the transfer of the data of said transfer source to said transfer targetaccording to the instruction; and wherein said logical device managementmeans updates said device configuration information after the completionof said data transfer.
 4. The storage system according to claim 3,further comprising external device monitoring means for determining saidexternal devices to be accessed at predetermined periods, and forexecuting input/output processing for said external devices.
 5. Astorage system, comprising: a first storage system; and a second storagesystem coupled to said first storage system and to a computer, whereinsaid first storage system comprises external device provision means forproviding a physical device in said first storage system to said secondstorage system as an external device; and, said second storage systemcomprises: logical device management means for managing, as deviceconfiguration information associated with logical devices accessed by ahost computer, physical devices in said second storage system andexternal devices provided by said external device provision means;input/output processing means for performing processing, when aninput/output request is received from the host computer, to convert thereceived input/output request into input/output processing for saidphysical device or said external device based on said deviceconfiguration information, and to perform input/output processingaccording to said conversion result; anomaly detection means for judgingthe occurrence, during input/output processing for an external device bysaid input/output processing means, of an access fault or performancedecline for the external device; notification means for issuing, when anaccess fault or decline in performance has been detected by said anomalydetection means, notification of the detected anomaly and the accessedexternal device to the management server which manages said secondstorage system and said first storage system; and, data transfer meansfor performing, upon receipt of an instruction from said managementserver for data transfer specifying the transfer source and transfertarget, the transfer of the data of said transfer source to saidtransfer target according to the instruction; and wherein said logicaldevice management means updates said device configuration informationafter the completion of said data transfer, according to theassociations after transfer of physical devices and external deviceswith said logical devices.
 6. A computer system, comprising: at leastone first storage system; a second storage system coupled to said firststorage system or systems; and a management server coupled to said firststorage system or systems and said second storage system, wherein saidfirst storage system comprises external device provision means forproviding a physical device in said first storage system or systems tosaid second storage system as an external device; said second storagesystem comprises: logical device management means for managing, asdevice configuration information, physical devices in said secondstorage system and said external devices, in association with logicaldevices accessed by a host computer; monitoring means for monitoringsaid external devices; notification means for notifying said managementserver when prognostication of a fault is detected by said monitoringmeans; and data transfer means for transferring data according to a datatransfer instruction received from said management server, and forupdating said device configuration information according to the datatransfer; and wherein said management server comprises data transferinstruction means for selecting a data transfer range and transfertarget based on notification from said notification means, and forissuing a data transfer instruction to said second storage system,specifying the transfer range and transfer target, and instructingtransfer of the data of said data transfer range to said transfertarget.
 7. The computer system according to claim 6, wherein said secondstorage system further comprises: input/output instruction means forperforming processing, when an input/output request for a logical deviceis received from the host computer, to convert the received input/outputrequest into input/output processing for said physical device or saidexternal device based on said device configuration information, and toperform input/output processing according to said conversion result; andanomaly detection means for judging the occurrence, during input/outputprocessing for an external device by said input/output processing means,of an access fault or performance decline for the external device;wherein said notification means issues, when an access fault or declinein performance has been detected by said anomaly detection means,notification of the detected anomaly and the accessed external device tothe management server; wherein said management server further comprisestransfer source decision means for identifying, based on informationreceived from said second storage system for said detected anomaly andsaid accessed external device, storage areas in which faults may occurdue to the anomaly as the transfer source; and, wherein said datatransfer instruction means selects the transfer target from among thestorage areas of said first storage system or systems connected toitself or said second storage system, based on the data capacity of saidtransfer source determined by said transfer source decision means and onevaluation of predetermined performance and reliability levels, andissues a data transfer instruction to said second storage system,specifying said transfer source and said transfer target.
 8. Thecomputer system according to claim 7, wherein said monitoring meansdetermines said external devices for accessing at predeterminedintervals, and causes said input/output instruction means to executeinput/output processing for said external devices.
 9. The computersystem according to claim 6, wherein said monitoring means monitors thestates of said external devices, and upon detecting a prognostication ofa fault occurrence, said notification means notifies said managementserver of the details of the detected prognostication and the externaldevice being monitored; said management server further comprisesanalysis means for analyzing, upon receiving said notification from saidnotification means, the received notification details, and fordetermining as the transfer source the external device which would beaffected upon occurrence of the fault contained in the notification;and, said data transfer instruction means confirms the capacity of thetransfer source external device determined by said analysis means,selects a device as said transfer target from among the devices beingmanaged by said second storage system and other than the external devicewhich was accessed when said fault prognostication occurred.
 10. Thecomputer system according to claim 9, wherein external devices providedby said external device provision means are comprised by RAID groupsconfigured by physical devices of said first storage system or systems;said notification means issues notifications of the occurrence of faultsin addition to notification of said prognostications of faultoccurrence; and, said analysis means, upon receiving notification offault occurrence from said notification means, analyzes the receivednotification, identifies the RAID group to which the external device inwhich the fault has occurred belongs, and recovers the data of saidexternal device in which the fault has occurred using data held by otherdevices constituting said RAID group, while also setting the recovereddata as said transfer source.
 11. The computer system according to claim9, wherein said data transfer instruction means, upon determining thetransfer target, confirms the performance and reliability level of thetransfer source external device determined by said analysis means, andselects a transfer target which, in addition to said capacity condition,satisfies said performance and reliability level condition from amongthe devices being managed by said second storage system and other thanthe device for which said fault prognostication occurred.
 12. Thecomputer system according to claim 11, further comprising a thirdstorage system coupled to said management server, wherein, if whendetermining the transfer target there is no transfer target device beingmanaged by said second storage system which satisfies said capacitycondition or said performance and reliability level condition, said datatransfer instruction means selects a device from among said thirdstorage system which satisfies at least said capacity condition as thetransfer target, and issues an instruction to said second storage systemto register as an external device the device determined as said transfertarget in the second storage system.
 13. A management server formanaging a first storage system and a second storage system coupled tosaid first storage system, said second storage system receiving anaccess request for a logical device from a host computer and accessingone of physical devices in said second storage system and an externaldevice in said first system according to received access request, saidmanagement server comprising: a processor; and memory storing a programexecuted by said processor, wherein, according to the program stored insaid memory, said processor performs: upon receiving notification fromsaid second storage system indicating prognostication of a faultoccurrence in said external device, transfer source decision processingto identify as the transfer source the storage area affected by saidfault based on said information; and data transfer instructiontransmission processing to select the data transfer target from thestorage areas of said first storage system or said second storage systemmanaged by said management server, based on the data capacity of saidtransfer source and the predetermined performance and reliability levelof the transfer source, and to transmit an instruction to said secondstorage system to transfer the data of said transfer source to saidtransfer target.
 14. A storage system, coupled to another storagesystem, which receives an access request to a logical device from a hostcomputer and accesses an external device in said other storage systemassociated with said logical device according to said received accessrequest, comprising: a plurality of physical devices, each associatedwith a logical device accessed by a host computer; a processor coupledto said plurality of physical devices; and memory storing a programexecuted by said processor and device configuration informationindicating the associative relation between said plurality of physicaldevices and said external devices with the logical devices accessed by ahost computer, wherein, in accordance with said program stored in saidmemory, said processor executes: input/output processing, upon receivingan input/output request from the host computer, to convert the receivedinput/output request into an input/output request for said physicaldevice or said external device based on said device configurationinformation, and to perform input/output processing according to saidconversion result; anomaly detection processing to judge the occurrenceof access faults or declines in performance for the external deviceduring input/output processing of said external device; notificationprocessing, when an access fault or decline in performance is detectedduring said anomaly detection processing, to notify said storage systemand a management server which manages said other storage system of thedetected anomaly and external device being accessed; data transferprocessing, when a data transfer instruction specifying a transfersource and transfer target is received from said management server, toperform data transfer according to said instruction; and, configurationinformation update processing to update said device configurationinformation after the completion of said data transfer, according to theassociation after data transfer of physical devices and external deviceswith said logical devices.
 15. The storage system according to claim 14,wherein said processor further performs input/output processing for saidexternal devices at predetermined intervals.
 16. A fault avoidance andrecovery method, in a computer system comprising at least one firststorage system, a second storage system, and a management server coupledto said first storage system or systems and to said second storagesystem, wherein said second storage system comprises logical devicemanagement means to manage, as device configuration information,physical devices comprised by said second storage system and externaldevices comprised by said first storage system or systems, inassociation with logical devices provided to a host; said methodcomprising the steps of: said second storage system, upon receipt of aninput/output request from said host, converting the receivedinput/output request into an input/output request for said physicaldevices or said external devices, based on said device configurationinformation; said second storage system performing input/outputprocessing according to the conversion result; when the device for saidinput/output processing is said external device, said second storagesystem judging whether an access fault or decline in performance hasoccurred for said external device during said input/output processing;when, as a result of said judgment, said access fault or decline inperformance is detected, said second storage system notifying saidmanagement server of the detected anomaly and the external device beingaccessed; said management server receiving said notification from saidsecond storage system; said management server specifying storage areasin which a fault may occur due to said anomaly as the transfer source,based on information contained in the received notification; saidmanagement server selecting a transfer target from among the storageareas of said first storage system or systems or said second storagesystem connected to said management server, based on the data capacityof said transfer source and predetermined performance and reliabilitylevels; said management server issuing an instruction to said secondstorage system to transfer the data of said transfer source to saidtransfer target; said second storage system, upon receiving the datatransfer instruction from said management server specifying the transfersource and transfer target, performing data transfer to transfer thedata of said transfer source to said transfer target according to saidinstruction; and, after the completion of said data transfer, saidsecond storage system updating said device configuration information,according to the associations after transfer of physical devices andexternal devices with said logical devices.
 17. The fault avoidance andrecovery method according to claim 16, wherein said external devices arecomprised by RAID groups configured by physical devices of said firststorage system or systems; when a fault occurrence is detected, saidsecond storage system issues notification of the occurrence of the faultand the external device in which the fault occurred to said managementserver; and, upon receiving notification from said second storage systemof the occurrence of the fault and of the external device in which saidfault has occurred, said management server recovers the data of theexternal device in which said fault has occurred, using the data held byother devices of the RAID group to which the external device in whichsaid fault has occurred belongs, and selects the recovered data as saidtransfer source.